It is known to render a volumetric imaging data set to produce a rendered image in which a subject of the image appears to be three-dimensional. The volumetric imaging data set may comprise a three-dimensional array of voxels, each with an associated intensity. The rendered image may comprise a two-dimensional array of pixels. The rendered image may sometimes be referred to as a 3D image because of the three-dimensional appearance of the subject in the rendered image.
In the context of medical imaging, a volumetric imaging data set may be obtained directly from a medical imaging scan or through further processes such as reconstruction and filtering. The intensities of the voxels of the volumetric imaging data may correspond to physical properties of the subject of the scan, for example tissue types. The intensities may be mapped on to opacity and color values, for example by using a transfer function. The opacity and color values may then be used in the rendering process, for example to distinguish different tissue types by rendering them in different colors.
Lighting effects may be added to a 3D image such that the subject of the image appears to be illuminated from a given position and/or direction. In recent years, 3D medical images have been made more realistic through the use of advanced lighting techniques (referred to as global illumination, gradient free lighting, subsurface scattering or photon mapping) that simulate illumination with a more physically accurate model than was previously used.
In Global Illumination (GI), a lighting model may be used that includes both direct illumination by light coming directly from a light source and indirect illumination, for example illumination by light that has been scattered from another surface.
A global illumination imaging algorithm may be divided up into two passes. The first pass is a forward simulation of light into the volume. Light information is stored in an intermediate volumetric structure, which may be referred to as a light volume or irradiance volume. The second pass traces light backwards from a camera. The second pass uses the light information in the light volume to render an image for display.
A two-pass system may allow a single light volume to be used for multiple rendering passes at different viewing angles, thus avoiding the need to recalculate the light volume for each render.
In an example of a GI imaging algorithm, the position of at least one light source is determined with reference to a volumetric imaging data set. In a first pass, virtual light is cast from the at least one light source into the volumetric imaging data set using a global illumination lighting model. The virtual light may be referred to in terms of photons or photon rays. The first pass may be referred to as a photon mapping pass.
The irradiance due to the at least one light source is determined at each of a large array of points in the volumetric imaging data set using absorptive properties assigned to the voxels in dependence on the voxel intensities. The irradiance values at the array of points are stored as a light volume, which may be also be referred to as an irradiance volume. The irradiance volume is representative of light cast into the volumetric imaging data set. The irradiance of each voxel in the light volume is determined by the number of photons that passes through the voxel.
The position of a viewpoint (which may be referred to as a camera) is determined. In a second pass, rays are cast from the camera into the irradiance volume (for example, one ray for each pixel of the resulting rendered image). Irradiances from points along each ray are integrated to provide pixel color values for a final rendered image.
Many global illumination algorithms use stochastic lighting simulation in an attempt to reduce or eliminate structured artefacts. In the first pass of the global illumination method, photons may be sent into the volume in accordance with a stochastic distribution. Optionally, the second pass of the global illumination method may also use stochastic simulation.
In some circumstances, stochastic simulation may introduce grainy noise into an image, which may result in a speckled appearance of the final image. Grainy noise may be unpleasant for a viewer. In some circumstances, grainy noise may make it more difficult for a viewer to interpret an image. For example, in a series of images that show a moving subject, the positioning of the grainy noise may shift from one image to the next in an unpredictable manner, which may be distracting to the viewer.
Grainy noise may be particularly prevalent in areas of an image that correspond to regions of the irradiance volume that were intersected by a relatively low number of photon rays. For example, some regions may be intersected by a low number of photon rays due to shadowing. Some regions may by intersected by a low number of photon rays due to scattering effects, for example multiple scattering.
One method of reducing grainy noise is to run the lighting simulation for a large number of iterations. FIGS. 1a, 1b and 1c are renderings of the same volumetric imaging data set using a different number of photon mapping passes. FIG. 1a uses one photon mapping pass. FIG. 1b uses two photon mapping passes. FIG. 1c uses 32 photon mapping passes.
In each photon mapping pass, photons of virtual light are distributed in a uniform but stochastic pattern across the footprint of the volume projected into the space of the light. The number of photon rays that are sent into the volume scales with the number of photon mapping passes. It may be expected that over multiple passes, the number of photon rays increases even in regions for which the number of intersecting photon rays is relatively low.
It may be seen from FIGS. 1a, 1b and 1c that the amount of grainy noise in the image decreases with an increased number of photon mapping passes. FIG. 1b has less grainy noise than FIG. 1a. FIG. 1c has less grainy noise than FIG. 1b. Each of FIGS. 1a, 1b and 1c comprises an insert showing an enlarged portion of the image, to better show the grainy noise.
It may be possible to reduce the grainy noise by running a number of iterations, for example until the result converges. However, running a large number of iterations may not be practical with regard to time and/or cost. Furthermore, it may be the case that a large proportion of the information available is obtained in the first iteration or first few iterations. Increasing the number of iterations may in some circumstances incur additional time and cost without resulting in much additional information.
Depending on stochastic parameters, the radiance reflected to the camera may be seen as samples of a radiance distribution. Filtering may be used as a tool for smoothing the distribution.
In many algorithms, filtering is performed as a post-processing pass on the rendered image. Such filtering is performed on the two-dimensional array of pixels of the rendered image rather than on a three-dimensional volume. Filtering of the two-dimensional array of pixels of the rendered image may be fast. Filtering of the rendered image may mean that the filtering only operates on the parts of the data that are visible in the image. Filtering only the relevant parts of the data may increase speed and/or decrease complexity of filtering.
It has been found that in some circumstances filtering of the rendered image may be too indiscriminate. In filtering, it may be desirable to remove noise while keeping meaningful features, for example sharply defined structures. It has been found that, in some circumstances, filtering of the rendered image may include filtering (for example, smoothing) of areas in which sharp structures are expected. The appearance of such sharp structures may be smoothed by the filtering in an unwanted manner.
In some circumstances, edge-aware filtering methods may be used to attempt to reduce the unwanted filtering of desired structures. However, in some circumstances even edge-aware filtering methods may filter areas in which filtering has an unwanted effect.
In some circumstances, a non-local means filter may work quite well. However, a non-local means filter may produce a patchy behavior where lighting quality is low.
Filtering of a three-dimensional volume instead of a two-dimensional rendered image is possible. In some methods, the three-dimensional irradiance volume may be filtered. However, in some circumstances, it may be difficult or impossible to filter a three-dimensional volume in real time. For example, a so-called 4D ultrasound image may be updated many times per second. It may not be possible to filter a three-dimensional volume quickly enough to apply such filtering to real time 4D ultrasound images. Some complex filters may take multiple seconds to render a single image.
Furthermore, volumetric filters may be more challenging than two-dimensional filters, especially since the photon data structure (the irradiance volume) may be complex.
FIGS. 2a, 2b and 2c provide a comparison of the same rendered image when different post processing filters are applied to the two-dimensional array of pixels in the rendered image. No filtering is applied to the image of FIG. 2a. Grainy noise is present in the image of FIG. 2a. A medium profile filtering is applied to the image of FIG. 2b. A high profile filtering is applied to the image of FIG. 2c. The image of FIG. 2c is more strongly filtered than the image of FIG. 2b. In the examples of FIGS. 2b and 2c, the filter used is an anisotropic diffusion filter (which is in the category of feature preserving image filters).
It may be seen that, while the filtering reduces the grainy noise, it also reduces a level of detail in the image. For example, a vessel 10 is shown in all three images. The vessel 10 appears less sharp in FIG. 2b than in FIG. 2a. The vessel 10 appears less sharp in FIG. 2c than in FIG. 2b. In the example of FIGS. 2a, 2b and 2c, the definition of detailed features (for example, the vessel 10) is lost before the grainy noise is fully eliminated. In general, definition may be lost before noise is fully resolved. In some circumstances, the noise magnitude may be quite high.